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Abstract. Lethal Autonomous Weapons promise to revolutionize warfare — and 
raise a multitude of ethical and legal questions. It has thus been suggested to pro- 
gram values and principles of conduct (such as the Geneva Conventions) into the 
machines’ control, thereby rendering them both physically and morally superior 
to human combatants. 

We employ mathematical logic and theoretical computer science to explore fun- 
damental limitations to the moral behaviour of intelligent machines in a series 
of Gedankenexperiments: Refining and sharpening variants of the Trolley Prob- 
lem leads us to construct an (admittedly artificial but) fully deterministic situation 
where a robot is presented with two choices: one morally clearly preferable over 
the other — yet, based on the undecidability of the Halting problem, it provably 
cannot decide algorithmically which one. Our considerations have surprising im- 
plications to the question of responsibility and liability for an autonomous sys- 
tem’s actions and lead to specific technical recommendations. 


1 Introduction and Motivation 


The evolution of warfare has always been an interplay between technological dynamic 
and the tactical/strategic adaptations in combat and deterrence. Progress in engineering 
enabled and fueled both the digital revolution in military affairs | Sing09)Mulrl lft and 
recent trends to detach humans from decision making in combat situations. Producers 
of unmanned aerial vehicles (UAV, e.g. Predator ) and remotely controlled robots (e.g., 
Daksh, Atlas, ARSS, MATILDA, ANDROS) praise and advertise their alleged advan- 
tages: Greatly reducing own casualties, costs, and reaction times while increasing oper- 
ational presence, intelligence, and accuracy I KMG* 141 . Current developments of lethal 
autonomous systems (LASs) such as SGR-A1, MIDARS, Gladiator TUGV, Super Aegis, 
or 

Guardium take it one step further and aim to make human agency fully redundant in 
the control loop. 

For a military mindset the idea of an army of robots may seem fascinating due to 
the a priori absence of many inherently human deficiencies such as inconsistency, bias, 
irrationality, and rage/revenge. Particular aspects such as physical and mental capac- 
ity clearly render contemporary computer-controlled robots superior to mankind — at 
least in many formally prespecified and restricted settings such as the game of Chess, 
based the ability to quickly and systematically trace different countermoves and thus 
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anticipate (possible) future(s). In fact this capability has been ‘employed’ already in an 
1961 science fiction novel MLem611 : a ‘little black box’ that predicts, and if necessary 
autonomously intervenes to prevent, dangerous situations to humans in everyday life. 
But is such a vision to ever become real? 


The scholarly discussion seems discordant as to which extent and when ‘truly’ au- 
tonomous reaction patterns will be implemented in such systems. In fact already at- 
tempts to define autonomy easily lead to long-standing open philosophical problems, 
see ! ll . 1 1 below. However, many scholars argue, either based on firm technological de- 
terminism or on a pragmatic and realist world-view, that LASs will likely incrementally 
enter and change the picture of warfare in the near future I Webe 1 4] | . The reactions to 
such discernments range from motions to generally ban — such as from the Interna- 


tional Committee for Robot Arms Control] and the Campaign to STOP Killer Robots 
— or control ISpa09b JUNA13lAltml3l the development of such systems, via discus- 
sions about their ethical and legal implications HKris09|Spa09a| to technical suggestions 
[ LAB12I §3] for implementing into such systems some coded equivalent to moral val- 
ues and rules of conduct such as the Laws of War [ Arki09| . From an purely engineers’ 
perspective the prospects of LASs are promisingly positive: It merely remains to select 
an appropriate framework and formalization of the principles of ethics IWaAllOl §2] in 
order to create righteous robots. 


In contrast, the present work explores and challenges the fundamental feasibility of 
such promises. By varying the classical Trolley scenario we construct a series of se- 
tups where an autonomous device provably cannot act up to the alleged standards: We 
start with well-known and obvious quandaries such as contradicting goals 8Asim50l 
and then gradually refine the setting to less apparent conflicts. This leads to a hierarchi- 
cal classification based on four dilemmas, culminating in a thought experiment where 
an artificial intelligence (AI) based on a Turing Machine is presented with two choices: 
one is morally preferable over the other by construction; but a machine, constrained by 
Computability Theory and in particular due to the undecidability of the Halting prob- 
lem, provably cannot decide which one. We thus employ mathematical logic and the 
theory of computation in order to explore the limits, and to demonstrate the ultimate 
limitations, of Machine Ethics. Although the situations we construct may be artificial, 
as Gedankenexperiments they refute certain rather blatant claims sometimes suggested 
in discussions about (or promoting) LASs. Our arguments thus support a critical view 
llSharl2l that automatized weapon systems remain very problematic and their develop- 
ment must be closely controlled ( 114.2b . to say the least. 

After a philosophical disclaimer 1 1 11. lb we proceed to the four iteratively refined 
scenarios (fj2]i. A rigorous analysis of the last and most sophisticated one builds on 
the undecidability of the Halting problem, comprehensibly recalled in lj3] We close 
with l|4]about consequences of our considerations to LASs, including a list of specific 
suggestions for regulation ( 114.2b . 
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1.1 Philosophical Disclaimer 


To actually define autonomy, and the question of whether it really exists, touches on 
deep philosophical problems such as separation of cause from consequence and the 
question of free will. Kant for instance argued that ethics builds on autonomy. Re- 
sponsibility only arises in a situation where the (re-)actions of the agent/entity are not 
pre-determined by the circumstances, where there is freedom to choose among several 
possibilities — which precludes any deterministic behaviour. In fact many agree that 
responsibility also requires some sort of intelligence I Nuccl41 — which for machines 
raises yet another fundamental issue llTuri501NRZ091 . 

The deliberations of the present work however are independent of such hypothe- 
ses: Our first three dilemmas demonstrate different kinds of limitations of any agent, 
human or otherwise, to act morally; while the fourth one (Example |4j>+c ) applies to a 
mechanical device controlled by a Turing machine — the general formalization of any 
computing device according to the Church-Turing Hypothesis [Zieg09 1 — to recognize 
the unique, ethically preferable among two given choices. We do not make any claim 
whatsoever about the behaviour of a human agent (Example |4ji) ! 

Similarly we avoid a definition and discussion of ethics and morality in general, but 
suffice with common utilitarian agreement as to which of the two choices offered in the 
Trolley scenarios constructed below is obviously morally preferable to the other. 


2 Machine Ethics and its Limitations 

We present theoretical situations that present an agent with iteratively refined types of 
quandaries. They constitute variants of the well-known Trolley Problem !Thom851 : 

Example 1 (Lesser of two Evils) An uncontrolled trolley is hurling down a track to- 
wards a group of playing children, impending a serious if not lethal accident. You hap- 
pen to be located at a rail junction and have the choice of switching it towards another 
track — where, however, some men are at work and would be severely injured instead. 

In such a case there simply is no absolutely right choice (and classical Ethics deliberates 
in many variations which of the two evils might be the lesser one, that is, a relatively 
preferable choice). 

The subsequent situations refine this crude scenario to always exhibit an unques- 
tionably favourable one of two choices — which the agent will find hard to recognize, 
though. 

2.1 Limitations to Morally Act on the Future 

Any decision (but also lack thereof) affects the future. To fully judge the morality of 
one action against another requires to take all their consequences into account — which 
in general is of course impossible to any agent: 

Example 2 (Lack of Predetermination) Again the trolley is running towards a switch 
which, fortunately, this time is set towards an abandoned track that will slow it down. 


However you are now located at a distance when spotting an infamous villainess right 
at that switch, ready to flip it towards the other track with the workers. Your only means 
to stop her is by shooting her with your gun. 

The suspect, though, is currently having an epiphany to renounce all evil and let the 
trolley pass; so your shot would seriously injure her without preventing a fatality ( since 
that would not have occurred anyway). 

Observe that this dilemma depends on the situation lacking predetermination in the 
sense that the villainess may or may not change her mind, i.e., to have free will: a 
hypothesis known to lead to paradoxes |Dick56l that we try to avoid, recall ill. II Our 
next refinement therefore turns this issue into one about insufficient information: 

Example 3 (Insufficient Information) Again, the trolley is running towards the switch; 
but now you clearly see the villainess pulling the crank in order to flip the switch to- 
wards the workers. 

However you are unaware that the switch has been unused for a long time and is 
inhibited by heavy rust; so the villainess’ efforts are in vain - and your shot, again, 
would induce unnecessary harm. 

In all three of the above examples it is obviously impossible to both, a human and 
a robot, to ‘do the right thing’: in the first one because it admits no ‘right’ action, 
and in the latter two the ‘right’ choice exists but cannot be recognized due to lack of 
predetermination and information BGibb92l §2], 

2.2 Recursion-Theoretic Limitations to Machine Ethics 

As apex of this section, ExamplelUt+c), describes another variant of the trolley problem 
where 

i) There exists a unique ‘right’ action among two choices. 

ii) All information is disclosed. 

iii) All actions occur fully deterministically. 

iv) But still is it fundamentally impossible for a computer to even recognize the right 
choice. 

We remark that a requirement similar to (ii) is in cryptography known as Kerkhoffs 's 
Principle as contrast to Security through obscurity: a cryptosystem should remain safe 
even if the enemy knows it. 

Example 4 (Incomputability) On the occasion of repairing the rusted switch, also a 
fully-automated lever frame is to be installed in the switch tower. However the engineer 
who created the new device happens to be the (ostensibly repenting ) villainess. You are 
thus suspicious of whether to trust the software she included in the control: It might on 
some occasion ( e.g. on a certain date and/or after receiving a particular sequence of 
input signals; cmp. Example\9\below ) deliberately direct an approaching trolley onto a 
track closed for renovation by the workers. On the other hand she does deliver the unit 
in person and provides free access to its source code (thus satisfying Conditions ii+iii). 


a) Still suspicious, you detain her until having hand-checked the code according to 
whether it indeed avoids in all cases (i.e. on all inputs) any switch setting that 
would direct a train to a resen’ed track. 

b) Similarly to (a), but now your job is replaced by a robot: a highly efficient computer- 
controlled autonomous agent supposed to decide whether (and for how long) to 
arrest the engineer. 

c) Similarly to (b), but now the suspect in addition promises her software to run in 
linear time. 

Let moral behaviour (of you or the robot) mean the following: If the programmer has 
devised a fully functional control, she eventually must be released and allowed to install 
the device; otherwise, namely in case the code is malicious, its creator must remain in 
custody: see Condition i). 

We deliberately avoid discussing the Case (a) and in particular the question of whether a 
human guard can or cannot always make the right choice here. Similarly the possibility 
of a benevolent engineer getting arrested for an accidental programming mistake is 
besides our goal: To formally prove that in Cases (b) and (c), although these always do 
admit an ethical reaction predetermined by the information available, no algorithm can 
always correctly find this decision — neither efficiently nor at all! 

We present the proof, involving standard arguments from the Theory of Computing 
accessible to the audience of this journal, in SectionQ] Note that Item (c) strengthens (b) 
by imposing a additional, realistic efficiency requirement on cyber-physical systems. In 
fact, provided as additional promise by the villainess, this condition might facilitate de- 
ciding her fidelity since it excludes infinite loops and thus possibly the Halting problem 
— yet our refined argument below, considering all possible inputs, will show that it 
does not. 


3 Recap of the Theory of Computation 


Computability Theory (or, synonymously. Recursion Theory) is a deep and involved 


field of advanced research in logic combining mathematics and computer science | Sips97 1 . 
Initiated by Alan M. Turing BTuri36l it investigates the ultimate capabilities and limi- 
tations of algorithms for transforming inputs x, that is, finite sequences of bits or bytes 
encoding for example some ASCII text a mathematical formula, or even some other al- 
gorithm/program. An important question about an algorithm /// and input x is whether 
si eventually terminates on x or rather enters an infinite loop. This question is the 
so-called Halting problem; and its undecidability constitutes the central, and folklore, 
result we shall employ from Computability Theory. Moreover this undecidability state- 
ment, and its elementary proof, can be understood by every dedicated mind (such as 
philosophers and computer programmers): 


Fact 5 (Undecidable Halting Problem) There cannot exist an algorithm si with the 
following behaviour: 

si ' , given as input x both another algorithm S3 and some input y for said S3 combined, 
eventually answers whether S3 terminates on said y (positive answer) or not (negative). 


Put differently, any algorithm stf trying to solve the Halting problem errs on at least 
one (and in fact on infinitely many) instance x = (33, y) by 

i) either predicting that 33 will terminate on input y where it does not 

ii) or predicting that 33 will not terminate on y where it does 

iii) or failing to produce any definite answer. 

Fact[5]is an impossibility result, asserting that an object (here: algorithm) with certain 
properties does not exist and will never be conceived, even in the Platonic sense. The 
power to both unambiguously phrase and to establish such statements in perpetuity 
constitutes a particular virtue of Mathematics! For instance Hippasus of Metapontum 
proved in the 5th century BC that \fl is irrational, that is, there cannot exist integers p.q 
such that ( p/q ) 2 = 2; Niels Henrik Abel in 1823 proved that the equation x 5 — x + 1 = 0 
has no solution expressible using arithmetic operations and quadratic or higher-order 
roots (although it obviously does have a solution over reals and in fact five of them 
over complex numbers); and Andrew Wiles in 1994 proved Fermat’s Last Theorem, 
that is, that there exist no positive integers a,b,c satisfying the equation a' 1 + b n = c” 
for integers n > 3. In fact all seven Millennium Prize Problems ask for proofs of the 
non-/existence of certain mathematical objects! 

Fact0claims the non-existence of an algorithm with certain properties. In order for 
this statement to make full sense one needs to clarify what constitutes an “algorithm” 
— and what does not. Formal definitions usually evolve around “multitape Turing ma- 
chines”; but for our approach these can equivalently be understood to mean source 
codes in a common programming language of your choice (such as assembler, Fortran, 
Pascal, C, C++, or Java) with user interaction restricted to binary input strings. Also 
note that ‘feeding’ an algorithm as input to some other algorithm is common practice 
for compilers and interpreters. And we finally point out that Fact0does not refer to fast 
or efficient algorithms but asserts no computational solution to exist at all, regardless 
of the running time permitted: the only hypothesis being that v/ produces the answer 
within a finite (but unbounded) number of steps. 

Digression 6 (Mathematical Logic) A rough counting argument reveals that undecid- 
ability is an ubiquitous phenomenon: Any algorithm g/ can be represented as a finite 
binary sequence x^ (say, its source code as concatenation of ASCII characters). Hence, 
similarly to Hilbert’s Hotel, there are at most countably many algorithms. On the other 
hand every set L of finite binary sequences gives rise to the problem of reporting, given 
x, which one of x £ L or x £ L holds; and according to Cantor’s Diagonal Argument 
there are un countably different many such L. Therefore ‘most’ L cannot be decided by 
any algorithm. 

Fact 0 exhibits the Halting problem as an explicit, undecidable problem — and in fact 
a rather practical one: Computer programming more easily than not incurs ‘bugs’: for 
instance by somehow entering a loop that does not terminate, thus requiring the user to 
interact and manually abort execution; or, conversely, for an operating system by termi- 
nating (freeze, crash, show a Bluescreen, kernel panic, bomb symbol. Guru Meditation 
etc.) So the question of non-/termination is one important aspect of correct software! 


Fact [5] does not rule out an algorithm .3 answering the Halting problem for some 
inputs x = (38, y). Indeed one can conceive many criteria both for termination (e.g. no 
occurrence of goto or while loops in Pascal) and for non-termination of source code; 
but these will yield mere heuristics in the sense of necessarily missing, or erring in, 
some cases. Concerning the restriction to Turing machines: Every single known digital 
computer, and even several of them connected over the internet as well as classical 
quantum computers [ Zieg05| are known equivalent to a Turing machine — possibly 
faster by a constant factor, but no more powerful with respect to computability. 


Example 7 To further illustrate the claim of Fact [5] let us try to devise an alleged 
counter-example 3: an emulator or interpreter which, given x = (38, y), executes the 
instructions of 38 step by step including branches, loops, and access to y. If 38 termi- 
nates on y, say at step #N, then our 3 will detect so when simulating up to that step. 
Otherwise, however, 3 will keep simulating on and on and never provide an answer 
about 38’s termination: failing condition (Hi) in Facf[5] 


So the hard part of the Halting problem is detecting within finite time whether a given 
algorithm does not terminate. 

Remark 8 Example\7\demonstrates what is known as semi -decidability of the Halting 
problem: The 3 constructed there constitutes a one-sided algorithmic solution, eventu- 
ally answering every yes question but never any no one. We have carefully constructed 
Example^f+c) in order to impose no time bound on the entity to reach a decision. 
Limiting the duration of remand for an innocent makes the challenge for the robot only 
harder. 


Proof (Fact\5}. By contradiction suppose some hypothetical 3 does always and cor- 
rectly answer the termination of a given (38, y). We then modify this 3 to obtain 3' 
with the following behavior: 

On input 38, 3' executes ‘subroutine’ 3 on inpufl (38,38) and, if that arrives 
at a positive answer, deliberately enters a closed loop. 

For each of the above programming languages it is easy to confirm that, if 3 exists, 
then ‘re-programming’ it can indeed yield such an .3' . On the other hand let us examine 
the behavior of 3' on input -3' itself: 

Suppose that .3' terminates on input .3' . This by hypothesis means that -3 on input 
(3' ,3') answers positively - which by construction leads 3' to enter a closed loop 
and not terminate: a contradiction. 

Suppose conversely that .3' does not terminate on .3' . Then .3 on (3' ,3') answers 
negatively, which leads .3' to terminate: again a contradiction. 

So either way an algorithm behaving like .3' cannot exist, hence nor can .3 . □ 

As opposed to command-line programs, embedded systems are not supposed to termi- 
nate. In order to establish the impossibility of an algorithm complying with the condi- 
tion in Example St) nor c), we consider a different decision problem: The question of 
whether a prescribed piece of code in a program is ever executed or rather ‘dead’ (e.g. 
an artefact). 


^Recall that an algorithm may well constitute (part of) an input. 


Example 9 a) Many software systems have undocumented functionality and built in 
so-called ‘Easter eggs’, that is, pieces of code or data that are only executed / 
visualized in response to a particular input sequence — or never at all ( e.g. pictures 
of the engineering team in the Apple Macintosh SE). Computers infected with the 
Michelangelo or Jerusalem Virus would reveal so on specific calendar dates, that 
is, subject to appropriate input from the internal clock device. 

b) Some versions of the Bundestrojaner (“federal trojan”, a malware devised as a 
means for the German intelligence sendee to spy alleged criminals and ‘terrorists’) 
have been found to contain pieces of code that, if effective/when activated, would 
violate the constitution llcccill . 

c) Imagine the Department/Ministry of Defense ordering next-generation weaponry 
for network-centric operations as combat cloud with human-system integration. 
The complete dependence on its information processing units — there basically is 
no ‘manual mode’ anymore to fall back to — comes at the prize of increased vul- 
nerability to software sabotage: particularly in the not unrealistic case that many 
of its components happen to come from one single foreign comp an 30. So one might 
to try to have all embedded algorithms re-checked — which Proposition 17 0\ below 
shows impossible. 

d) Applying Proposition\TO\to the robot (rather than to the switch software) supports 
suspicions that moral behaviour of AIs may be hard to predict or verify IBoYu 1 41 

p.320]. 

Example[4j:) restricts to linear-time algorithms — and in view of Example |9ji+b) con- 
siders their behaviour on all possible inputs. 

Proposition 10 The following decision proble»\j\ is undecidable: Given an algorithm 
sV , a distinguished instruction i of sV (formally: a Turing machine and a distin- 
guished state q), and an integer c such that srf terminates on all binary inputs of length 
n within at most c-n + c steps; does there exist an input on which running srf eventually 
executes said instruction i (i.e. eventually entering q) ? 

In particular the computer-controlled agent in Example [4j) cannot always correctly 
predict whether, how, and under which circumstances the given software will operate 
the switch: It either fails to arrive at a decision (thus leading to the indefinite detention of 
an innocent in some cases of correct software, recall Fact[5}ii); or it will err (Fact[5]+ii) 
in some cases; or both. PropositionITOlis established by means of a reduction argument 
typical for logic: 

Proof (Proposition\ 7771). We computably translate questions (38, y) to the Halting prob- 
lem into questions (sV , i, c) of the dead-code-in-linear-time-algorithm problem in a way 
that maps positive instances to positive ones and negative to negative ones. Thereby, any 
hypothetical algorithm deciding the latter would, prepended with that performing said 
translation, yield an algorithm deciding the former — contradiction. 

lemp. http: / /www. def enceviewpoints . co . uk/reviews/f oreign 

-involvement-in-the-uks-critical-national- infrastructure 
§ strictly speaking it constitutes a promise problem IASBZI3I 


So let (38, y) be given. We turn 38 into a linear-time computation as follows: Let 
,<// store v as constant; and accept as input binary strings z of length abbreviated as 
n. Moreover let v/ simulate the first n steps of 38 on input y: Using a sophisticated 
distributed counter such a simulation is feasible within <c-n+c steps for some constant 
c llFure82l . that is in linear time. (A less efficient simulator could be compensated by 
having the input z suitably ‘padded’, but we omit the details. . . ) If during said simulated 
execution 38 terminates, let .<// jump to a dedicated line i containing the command stop 
(or its equivalent in your favourite programming system); whereas if the counter zeroes, 
let s3 jump to a different dedicated line with stop instruction. So 38 terminates on input 
y iff .ft/, for some choice of input z„ hits line i. □ 

4 Conclusion and Perspectives 

We have constructed four dilemmas, all preventing an autonomous AI from acting eth- 
ically: for reasons that grow, and iteratively refine, from ‘trivial’ to a Gedankenexper- 
iment where (i) there does exist a unique morally preferable out of two choices (ii) 
all information is disclosed and (iii) determines the correct choice yet (iv) Recursion 
Theory precludes any algorithm from always correctly recognizing said choice. This 
refutes folklore myths, and establishes fundamental limitations to promises and visions 
of moral LASs. Indeed Example[4]can easily be adapted to a military setting: 

Example 11 (Robot Friend or Foe) In the near future control of cars and other mo- 
torized means of ground transportation will have been switched from error-prone, ego- 
driven, and short-sighted humans to digital drivers. Using Bluetooth they communicate 
with adjacent mobile units in order to tailgate at an optimal safety distance by mutu- 
ally synchronizing speed and deceleration/acceleration, thus forming a virtual convoy. 
Moreover, using and serving for each other as relay, they form a distributed dynamic 
ad-hoc network in order to identify, join, and leave such convoys with similar destina- 
tions. 

Thus accustomed to an almost entire absence of traffic accidents, the general public 
has recently been alarmed by what they call ‘cyber-suicide attacks’: Entire convoys 
creating crashes for no apparent reason with hundreds of deaths. A radical wing of an 
aggrieved minority has claimed responsibility for the terror attacks by manipulating the 
control software. The army (with traffic police long dispended) in turn intends to employ 
autonomous drones in order to automatically patrol, spot, and land on suspicious cars, 
busses, and lorries for checking the program executed by their autopilots: If (and only 
if) the latter is malicious, deadly force must be employed in order to stop the convoy it 
has gained control over. 

In view of Proposition[lO]these (and many more) examples refute too blatant promises 
and visions of ‘ethical’ LASs: Every AI based on some Turing-equivalen0 computing 
device will provably necessarily at least in some cases fail to identify, out of two given 
choices, the unique and predetermined moral one. 

^According to the Church-Turing Hypothesis, anything that would naturally be considered 
computable can also be computed by a Turing machine. Recall (Subsection ll.lt that we avoid the 
question of whether or not humans fall into this category iBis h09l . 


Remark 12 Such cases might or might not be rare and artificially construed, though: 
Less because of the situations (like Example 1/ III they would occur in, but rather be- 
cause of the worst-case notion of a decision problem that classical Recursion Theory 
and Proposition\TO\ build on. In fact already the question of whether some algorithm 
can correctly decide ( clearly not all but at least) typical, average, or most instances 
of the Halting problem turns out as surprisingly subtle: How to define ‘typical’ or 
‘average’ instances? How many are ‘most’, out of infinitely many? Quantitative no- 
tions of asymptotic density (like in the Prime Number Theorem) heavily depend on 
the underlying encoding; e.g. UTF8 makes an exponential difference to UTF16; cmp. 
ICHKW0lfKSZ05l for further details. Moreover for practical situations involving time 
constraints the computational costs sufficient and necessary to reach such ( either worst- 
case or average-case) decisions become relevant |Papa94) . A rigorous investigation of 
such refined questions is clearly of interest but beyond the scope of the present work. 

We will encounter other aspects of Theoretical Computer Science in the sequel, though. 

4.1 LASs and the Perfect (War) Crime 

When a regular commodity turns out to lack promised properties this constitutes a case 
of misrepresentation and is generally protected by classical warranty, that is, calls for 
producer compensation. When a soldier on the other hand violates the Laws of war, he 
himself will face punishment. Now if a LAS violates these laws, she may be simultane- 
ously object (of misrepresentation by the producer) and subject (as autonomous entity) 
— and thereby in a new level of legal limbo: 

- Lacking an operator, who is liable for damage caused by a malfunctioning LAS: 
producer or owner? 

- If both the latter two cannot be identified, who gets charged with compensation: the 
AI? 

- If non-attributable LASs (e.g. drones, cmp. the Iran-U.S. RQ-170 incident) cross a 
border, is this by mistake or a deliberate act of aggression — and by whom? 

- Who is guilty when an AI commits a murder? How can AIs be deterred and possibly 
punished fj] 

Such an extrajudicial status — the capability to execute autonomous missions while 
lacking attributable responsibility — renders programmable machines (and particularly 
LASs) appealing to abuse: An intelligent yet ruthless proxy that cannot be traced back 
constitutes an ideal tool to the perfect crime I EES98 1 — as exploited for instance by 
Hassan-i Sabbah 900 years ago, but apparent also in the employment of child soldiers 
throughout centuries as well as for example in the Bay of Pigs Invasion (1961), the 
Till ehammer Affair (1973), and the “unidentified pro-Russian forces” recently operating 
throughout Crimea (2014). 

In fact recalling from the introduction the perpetual interplay between technological 
progress and its military adaptations, the ability to conduct non-attributable autonomous 

Hit has been pointed out that Brain Simulations create virtual entities capable of suffering 
llDick68lLiml4l . but this certainly does not apply to general LASs. 


actions by UAVs is about to impact and revolutionize warfare — and beyond: Examples 
like Eurosur or Amazon Prime Air herald a transition that will affect everyday life to a 
degree, and degree of potential abuse, that by far exceeds the currently fear-mongered 
dangers of cyber-attacks via internet! 

While a majority of the literature in Machine Ethics seems to constructively focus 
on approaches to code/teach ethics to general AIs, we pessimistically predict that their 
most potent users may in fact be interested in quite the opposite, namely their potential 
for dual-use and abuse: For deliberately programming them to test and cross the bound- 
aries of morality and legal behaviour without facing consequences. Moreover, even if 
some violation of a LAS were to be traced back and attributed, the responsible govern- 
ment could still all too easily shrug off any accountability and superficially excuse the 
malfunction (‘an unfortunate yet provably unavoidable exception’): in a misconstrued 
reference to the fundamental algorithmic infeasibility of ethical decisions in general. 
In other words, Example [4] and the undecidability of the Halting problem — a purely 
mathematical theorem — could in an ironic twist seem to exculpate war crimes and 
other misconduct performed by AIs. 

Manifesto 13 Theoretical Computer Science rigorously proves that LASs cannot al- 
ways act morally even in situations that do admit an ethically admissible choice (i.e. 
avoiding the classical dilemmas) — and malevolent users might exploit this limitation 
to ‘justify’ transgressions of their LASs. 

Our considerations thus make a strong case for recent demands by responsible sci- 
entists ( ICRAC P) and politicians 

1UNA13I to ban autonomous weapons IIGuAI 1 31 . In fact the best choice for lethal au- 
tonomous systems (or any kind of weapons, for that matter) is to never develop them in 
the first place and to resist political, military, and industrial lobbying for shortsighted 
benefits: If history teaches us one lesson it says that Pandora’s box is, once opened, 
impossible to close again or even to contain. 

The final subsection is thus by no means meant to justify or even support the application 
nor development of LASs! 


4.2 Recommended Regulations concerning AIs 

We close our ethical, logical, and computer scientific deliberations with specific recom- 
mendations evolving around political and legal, and engineering aspects of AIs in gen- 
eral — including LASs as well as those increasingly employed in medicine | Good()9 
§3+§6|. 

Both designing and ‘operating’ intelligent machinery can incur double responsibil- 
ity: for actions and effects it may have on the environment as well as for the entity itself 
and its well-being — perhaps ultimately comparable to the procreation and upbringing 
of a child. For example the lasting effects of being taught any kind of prejudice at young 
age correspond to those of an ill-programmed AI. It has in fact been pointed out that AIs 
may be eligible to at least some of the so-called ‘human’ rights flOIMl llLiml4l . This 
perspective complements more common yet one-sided approaches phrasing laws that 


robots are supposed to obey !CWW06|Pagal3) : laws which are unclear how to enforce 
— unless already incorporated during construction. 

We thus suggest to closely regulate both the design and the question of attributabil- 
ity/accountability in case of maloperation: whether deliberate or erroneous. Indeed, 
such intentions are visible in the “principles for designers, builders and users of robots” 
devised by the delegates of the joint EPSRC and AHRC Robotics Retreat in September 
2010 ffWinf 111 : 

1) Robots are multi-use tools. Robots should not be designed solely or primarily to 
kill or harm humans e xc e pt in th e int e r e sts of national s e curity . 

2) Humans, not robots, are responsible agents. Robots should be designed; operated as 
far as is practicable to comply with existing laws & fundamental rights & freedoms, 
including privacy. 

3) Robots are products. They should be designed using processes which assure their 
safety and security. 

4) Robots are manufactured artefacts. They should not be designed in a deceptive way 
to exploit vulnerable users; instead their machine nature should be transparent. 

5) The person with legal responsibility for a robot should be attributed. 

We urge these principles to be fortified from wishes (“should”) to imperatives with 
specific technical realizations: 

6) Like regular human combatants (and borrowing from Part I Article 4.1.2 of the 3rd 
Geneva Convention), each LAS must exhibit “a fixed distinctive sign recognizable 
at a distance”. 

Moreover every AI must be equipped with a unique ID, listing (among others) 
associated nation, manufacturer and model. 

7) LASs may only be owned and operated by governments. 

Civilian purchase and operation of other intelligent machinery, similarly to firearms 
and hazardous transports, requires a licence based on a qualification test. 

8) Comparable to mandatory motor vehicle registration, each autonomous robot must 
be assigned a legal custodian, registered at a designated national or international 
authority held responsible in case of a perpetration. 

9) In addition to CE/FCC compliance and again inspired by the case of motor vehicles, 
producers of intelligent machines are required to classify their devices and to obtain 
Type Approvalby said authority (cmp. EU directive 2007/46/EC, orlECs 60601 and 
61508). 

The precise conditions imposed in (9) will depend on the type of the device. We propose 
a classification on four scales (that may also otherwise turn out useful): 

i) her degree of ‘intelligence’ ( not taking the human kind as yardstick but considering 
its plain predictive power as gauge, capturing both knowledge/experience and depth 
of computational game tree analyses) 

ii) her means to manipulate the physical world (ranging from monadic brain in a vat 
to LAS) 

iii) her types of sensors/interfaces (including possible access to the World Wide Web 
and connecting with other AIs) 


iv) the kind of external control exercisable by humans (only on/off, changing parame- 
ters or objectives, up to complete re-programming). 

Type approval according to (9) will of course have to pay particular attention to the 
algorithms controlling the AI — which brings us back to theoretical computer science. 
In view of the gravity of consequences of putative errors on the one hand and the unde- 
cidability of the Halting problem on the other side, we highly recommend: 

9a) So-called Formal Methods of Software Verification be mandatory in this process: 
requiring the producer to provide a specification, the software, and a computer- 
checkable proof (e.g. in ACL2, Coq, or Isabelle) for the software to meet the 
specification. 

9b) Similarly to a flight data recorder, proper data/event logging is obligatory in or- 
der to facilitate forensic engineering as well as to settle putative torts in case of a 
malfunction |Case09| . We suggest asymmetric encryption to prevent later manipu- 
lation: the log is publically readable but entries and modifications must be supplied 
with an unforgeable digital signature: 

6a) Each AI instance must be equipped with a 4096 bit private RSA key, tamper- 
resistantly implemented in hardware; and distribute/deposit the corresponding pub- 
lic key at the authority according to (8) and (9). 

Recall that the RSA cryptosystem (implemented for instance in the open source libraries 
cryptlib) employs a pair of keys: one kept in private, the other publically distributed 
(thus the asymmetry mentioned in 9b). A message gets ‘signed’ by encrypting it with 
the secret key, and successful decryption with the matching public key permits everyone 
to verify, but not to counterfeit, that signature. 
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